Skip to content

SLO: transactional topic<->table workloads (sync/async), exactly-once under TLI - #854

Open
vgvoleg wants to merge 3 commits into
mainfrom
slo-topic-tx-transactions
Open

vgvoleg wants to merge 3 commits into
mainfrom
slo-topic-tx-transactions

Conversation

@vgvoleg

@vgvoleg vgvoleg commented Jul 3, 2026

Copy link
Copy Markdown
Member

What

Adds two SLO workloads — sync-topic-tx / async-topic-tx — that validate the both-ends transactional topic ↔ table pipeline (exactly-once) under chaos and TLI.

  • Producer (table → topic): tx_writer(tx) writes a batch to the topic while the same tx does a read-modify-write on a shared hot row. Commit persists the topic write + table update atomically; seqno advances only on commit (no gap on abort).
  • Consumer (topic → table): receive_batch_with_tx(tx) reads a batch while the same tx UPSERTs each message into a sink table keyed by (writer_id, seqno) and bumps a hot row. Commit advances the offset + writes rows atomically → a rolled-back tx re-reads and re-UPSERTs the same keys (idempotent, exactly-once).
  • TLI on purpose: both txs contend on a few shared hot rows (--tli-hot-keys) → ydb.Aborted (locks invalidated). retry_tx_* absorbs it (counted via on_ydb_error_callback → topic_tx_tli, neutral). Even retry-exhaustion recovers with no loss.

Reuses the existing topic writer/reader machinery, payload codec, metrics, and reconnect/recreate loops. Design doc: tests/slo/TOPIC_TX_SCENARIO.md.

Verified (local /local, 30s each)

TLI absorbed Lost jobs start/stop sink
sync 115 0 4+4 clean 1079 rows, seqno gap-free 1..max
async 95 0 4+4 clean 1205 rows, seqno gap-free 1..max

black / flake8 / py_compile clean. No CHANGELOG (internal tests/CI).

… under TLI

Both-ends transactional pipeline: tx_writer (table->topic) and
receive_batch_with_tx + sink upsert (topic->table), commit makes topic
offset and table write atomic. TLI is induced via hot-row contention and
absorbed by retry_tx; run stays exactly-once and loss-free.
@codecov-commenter

codecov-commenter commented Jul 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.22%. Comparing base (a02e73f) to head (d82edb6).
⚠️ Report is 69 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #854   +/-   ##
=======================================
  Coverage   81.22%   81.22%           
=======================================
  Files          94       94           
  Lines       12111    12111           
  Branches     1184     1184           
=======================================
  Hits         9837     9837           
+ Misses       1809     1808    -1     
- Partials      465      466    +1     
Flag Coverage Δ
integration 79.00% <ø> (ø)
unit 47.37% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds new SLO workloads (sync-topic-tx / async-topic-tx) to validate an exactly-once transactional pipeline between YDB Topic and Table under chaos + deliberate TLI contention, integrating the new metric (topic_tx_tli) into SLO reporting and CI.

Changes:

  • Implement transactional topic↔table producer/consumer job managers (sync + async) using tx_writer / receive_batch_with_tx, plus runner/CLI wiring.
  • Add topic_tx_tli metric emission and expose it in metrics/threshold configs.
  • Document the scenario and update SLO README / workflow matrix / entrypoint ref-scoping for tx topics + tables.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/slo/TOPIC_TX_SCENARIO.md New design doc describing the transactional topic↔table SLO scenario and invariants.
tests/slo/thresholds-topic.yaml Adds topic_tx_tli as a neutral (informational) threshold metric.
tests/slo/src/runners/topic_runner.py Creates/drops tx sink+hot tables for *-topic-tx workloads; selects tx vs non-tx job managers.
tests/slo/src/options.py Adds CLI options for sink/hot tables, hot-key count, and messages-per-tx.
tests/slo/src/jobs/topic_tx_jobs.py New sync transactional workload implementation (topic→table + table→topic in one tx).
tests/slo/src/jobs/topic_jobs.py Refactors validation to reuse _record() for both classic and tx workloads.
tests/slo/src/jobs/async_topic_tx_jobs.py New async mirror of the transactional workload.
tests/slo/src/jobs/async_topic_jobs.py Refactors validation to reuse _record() for both classic and tx workloads.
tests/slo/src/core/metrics.py Adds inc_tli() and OTLP counter sdk.topic.tx.tli.total.
tests/slo/README.md Documents new *-topic-tx workload labels and describes behavior/options.
tests/slo/metrics-topic.yaml Adds PromQL query/series for topic_tx_tli.
tests/slo/docker-entrypoint.sh Adds ref-scoped topic + sink/hot table naming for *-topic-tx workloads.
.github/workflows/slo.yml Runs new sync-topic-tx / async-topic-tx workloads in the SLO workflow matrix.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/slo/TOPIC_TX_SCENARIO.md Resolved
Comment thread tests/slo/thresholds-topic.yaml Resolved
Comment thread tests/slo/metrics-topic.yaml Resolved

@robot-vibe-db robot-vibe-db Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Summary

Verdict: ✅ No critical issues found

Critical issues

No critical issues found.

Other findings

  • Nit | High: Non-English word "штатный" in documentation and config comments — TOPIC_TX_SCENARIO.md:12, thresholds-topic.yaml:33, metrics-topic.yaml:55
  • Minor | Low: Table names interpolated into SQL via .format() (standard for YDB, but worth noting as a code smell for maintainability) — topic_tx_jobs.py:56-58, topic_runner.py:59-67

Overall assessment: This is a well-designed PR. The transactional semantics are carefully handled:

  • seqno advances only after successful commit (no gap on abort)
  • The committed list is reset on retry via nonlocal (correct for retry_tx)
  • Sink UPSERT is keyed by (writer_id, seqno) making re-processing idempotent
  • Hot rows are correctly seeded and _bump_hot gracefully handles missing rows
  • The _record refactor in parent classes is clean and preserves existing behavior
  • Thread safety is correct: sync uses _expected_lock, async relies on single event loop
  • The _validate → _record split correctly separates message decoding from delivery tracking, allowing tx subclasses to call _record only after commit
  • Ref-scoping of topics and tables prevents cross-contamination between current/baseline containers
  • The entrypoint correctly sanitizes ref names and scopes all resources

This review was generated automatically. Critical issues require attention; other findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.

Comment thread tests/slo/TOPIC_TX_SCENARIO.md Resolved
self.messages_per_tx = max(1, int(getattr(self.args, "messages_per_tx", 10)))
self.hot_keys = max(1, int(getattr(self.args, "tli_hot_keys", 4)))

self._sink_upsert_query = SINK_UPSERT_TEMPLATE.format(sink_table)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Low

Table names are interpolated into SQL strings via .format(). This is the standard pattern in the existing SLO codebase (same approach in table_runner.py) and YDB doesn't support parameterized table names, so it's not exploitable here — especially since the inputs come from CLI args sanitized in docker-entrypoint.sh (via tr -c 'a-zA-Z0-9_' '_'). Just flagging for awareness since it's a new set of format-injected queries.

@robot-vibe-db

robot-vibe-db Bot commented Jul 3, 2026

Copy link
Copy Markdown

Full analysis log

Analysis performed by claude, claude-opus-4-6.

Remove the hot-row contention table, the tli_hot_keys/hot_table options, and
the topic_tx_tli metric. The scenario keeps its point: a transactional topic
write (tx_writer) and a transactional topic->sink-table consumer
(receive_batch_with_tx + upsert), exactly-once under chaos. Single sink table.
@vgvoleg vgvoleg added the SLO label Jul 3, 2026
@github-actions github-actions Bot removed the SLO label Jul 3, 2026
read_availability was ~90% (async) not because reads failed but because a
receive_batch_with_tx that blocks (idle reader, or the SDK transparently
reconnecting under chaos) hit our wait_for timeout, which is not a ydb.Error
so the tx retrier never retried it — and we wrongly recorded it as a failed
read. Add metrics.cancel(): a timeout or an empty batch now balances the
pending counter without a success/failure, so read_availability reflects the
success rate of actual transactional reads (which the retrier drives ~100%).
Real outages remain visible via write_availability / topic_delivered_rps.
@vgvoleg vgvoleg added the SLO label Jul 3, 2026
@github-actions github-actions Bot removed the SLO label Jul 3, 2026
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown

🌋 SLO Test Results

🔴 7 workload(s) tested — 2 workload(s) exceeded failure thresholds

Commit: b735e98 · View run

Workload Thresholds Duration Report
async-query 🟡 Warning 10m 3s 📄 Report
sync-table 🟢 OK 10m 2s 📄 Report
async-topic-tx 🔴 Failure 10m 3s 📄 Report
sync-query 🟢 OK 10m 11s 📄 Report
async-topic 🟢 OK 10m 3s 📄 Report
sync-topic-tx 🟢 OK 10m 3s 📄 Report
sync-topic 🔴 Failure 10m 7s 📄 Report

Threshold violations:

async-query:

  • read_retry_attempts: Value 0.01 > warning max 0

async-topic-tx:

  • topic_e2e_latency_p99_ms: ▲ 49.8% (≥ 30% warn)
  • read_latency_p95_ms: ▲ 34.1% (≥ 30% warn)
  • read_latency_p99_ms: ▲ 34.1% (≥ 30% warn)
  • write_retry_attempts: ▲ 300.0% (≥ 50% fail)

sync-topic:

  • read_retry_attempts: ▲ 50.0% (≥ 50% fail)

Generated by ydb-slo-action

@robot-vibe-db

robot-vibe-db Bot commented Jul 3, 2026

Copy link
Copy Markdown

Full analysis log

Analysis performed by claude, claude-opus-4-6.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Comment on lines +23 to +31
"""Transactional topic <-> table pipeline (sync).

Producer: a ``tx_writer`` writes a batch of messages to the topic inside a
transaction (a transactional topic write). Consumer:
``receive_batch_with_tx`` reads a batch while the same transaction UPSERTs
each message into a sink table keyed by ``(writer_id, seqno)``. The commit
advances the topic offset and persists the sink rows atomically -> a
rolled-back tx re-reads and re-UPSERTs the same keys (idempotent), so the
table receives every message exactly once, even under chaos.
Comment on lines +71 to +76
def cancel(self, labels, start_time: float) -> None:
"""Undo a ``start()`` for an operation that turned out to be a no-op
(e.g. a transactional reader that had no batch to process): balances the
pending counter without recording a success or a failure, so idle waits
do not count against availability."""
return None

@robot-vibe-db robot-vibe-db Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review Summary

Verdict: ❌ Critical: 0; Major: 1

Critical and Major issues

  • Major | Medium: No explicit timeouts anywhere on the transactional path (retry_tx_sync/retry_tx_async, tx.execute, tx.commit, tx_writer.write); YDB's default gRPC deadline is 600 s, so one stuck RPC under chaos can block a writer/reader thread for up to 10 minutes — far beyond --time — while run_tests() joins these threads, stretching or stalling the whole SLO run — tests/slo/src/jobs/topic_tx_jobs.py:100

Other findings

  • Minor | High: The topic_duplicates invariant described in the scenario doc ("redelivery after a rolled-back tx") can never fire in the implemented flow, because _record runs only after a successful commit, so rolled-back re-delivery is always recorded as a fresh delivery — tests/slo/TOPIC_TX_SCENARIO.md:73
  • Minor | Medium: topic_e2e_latency is documented as "write-commit → read-commit latency" but write_ts_ns is captured at payload-encode time, before the producer transaction starts and before any retry backoff, so consumer-side e2e absorbs producer retry delays after chaos aborts — tests/slo/TOPIC_TX_SCENARIO.md:72
  • Minor | Medium: The TimeoutError → cancel() branch treats every receive timeout as an idle reader; a starved reader whose producers are all wedged in long (timeout-less) transactional writes also shows up as idle with read_availability ~100%, hiding the stall from the availability signal — tests/slo/src/jobs/topic_tx_jobs.py:149

This review was generated automatically. Critical and Major issues require attention; Minor and Nit findings are advisory.
If this comment was useful, please give it a 👍 — it helps us improve the review bot.


ts = self.metrics.start((OP_TYPE_WRITE,))
try:
pool.retry_tx_sync(callee, retry_settings=self.retry_settings)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Major
Confidence: Medium

The transactional path passes no timeout anywhere: retry_tx_sync(callee, retry_settings=self.retry_settings) runs a callee that calls tx_writer.write(...) (whose first write does a write_with_ack flush), and the pool wrapper then calls tx.commit(), both without any timeout; the consumer side (_run_topic_reads) likewise calls tx.execute for the sink UPSERT without settings. YDB's default gRPC deadline for calls without settings is DEFAULT_TIMEOUT = 600 s (ydb/connection.py), so under chaos a single RPC to a killed-but-TCP-alive node can block this writer thread for up to 10 minutes — far beyond --time — and run_tests() joins the job threads, so the whole run stalls with it. The plain topic jobs deliberately wrap every operation with write_timeout/read_timeout (see the "so the workload never silently hangs" comments in topic_jobs.py); this PR's tx jobs lost that protection.

Suggested fix: pass a bounded deadline into the transactional operations — e.g. build BaseRequestSettings(timeout=...) (from --write-timeout/--read-timeout) for tx.execute/tx.commit, and/or bound the whole retry attempt with asyncio.wait_for in the async manager the same way receive_batch_with_tx is bounded.

| `topic_lost_errors` | forward gap in a producer's seqno (real loss) | **0** — hard-fails the run, under chaos |
| exactly-once into sink | sink keyed by `(writer_id, seqno)` — idempotent UPSERT | re-processing after a rollback creates no duplicate row |
| `topic_e2e_latency_p50/p99` | write-commit → read-commit latency | reported |
| `topic_duplicates` | redelivery after a rolled-back tx | informational (at-least-once at the app level) |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: High

This invariant row documents topic_duplicates as "redelivery after a rolled-back tx", but that signal is unreachable in the implemented flow: _record is called only after retry_tx_* returns (i.e. only after the commit succeeded), so a consumer tx that rolls back is never recorded at all — its messages are re-read and recorded once as fresh deliveries (seqno == exp), never as duplicates. The duplicates counter in this workload can only fire on a different mechanism: the producer re-committing an identical seqno range after a lost commit response.

Suggested fix: reword the row to describe the producer re-commit duplicate case, or note that consumer-side rollback redelivery is invisible to topic_duplicates by design.

|--------|---------|-------------|
| `topic_lost_errors` | forward gap in a producer's seqno (real loss) | **0** — hard-fails the run, under chaos |
| exactly-once into sink | sink keyed by `(writer_id, seqno)` — idempotent UPSERT | re-processing after a rollback creates no duplicate row |
| `topic_e2e_latency_p50/p99` | write-commit → read-commit latency | reported |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

The table defines topic_e2e_latency_p50/p99 as "write-commit → read-commit latency", but the code captures write_ts_ns via time.monotonic_ns() inside encode_payload, which runs before the producer transaction starts (and before the limiter pacing loop completes for the rest of the batch). If a chaos abort forces retry_tx_* to re-run the callee, backoff (RetrySettings defaults grow to seconds per attempt over 10 retries) is absorbed into the consumer-side e2e measurement, so the metric drifts from its documented meaning and can inflate the gated topic_e2e_latency_p50 after aborts.

Suggested fix: either reword the doc to "encode → read-commit", or capture the write timestamp on the final attempt (e.g. right before tx.commit() / after the first successful write ack) so e2e measures what the doc claims.

# the SDK transparently reconnecting under chaos,
# not a read failure. Don't count the op (real
# errors surface as exceptions the tx retries).
self.metrics.cancel((OP_TYPE_READ,), ts)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: Minor
Confidence: Medium

Every receive timeout is classified as an idle no-op (cancel(), no success and no failure). That is correct for a genuinely empty topic, but it also covers the starvation case: if all producers are wedged in timeout-less transactional writes (see the missing-timeouts issue), readers sit here receiving nothing, and read_availability stays at ~100% with zero error counters — the availability signal cannot distinguish "no work to do" from "pipeline stalled". The plain topic reader intentionally counts a starved timeout as a read failure for exactly this reason (comment in topic_jobs.py).

Suggested fix: keep the no-op classification if desired, but make starvation visible another way — e.g. count consecutive empty timeouts per reader and emit them as a separate metric, or count a timeout as a failure after the reader has been idle for longer than the expected inter-batch interval.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants